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ABSTRACT 



The adoption of alternative pedagogical philosophies in the 
classroom has led to an increased use of technology to expand and enhance 
authentic, contextual learning environments. Correspondingly, these new 
approaches have also led to a growing dissatisfaction with existing 
evaluation methodologies to evaluate knowledge. This paper proposes such an 
evaluation model based on the premise that evaluation strategies should 
reflect the full range of the experiences of learning. The model incorporates 
evaluation strategies that provide a ..composite picture of learning by 
examining both the learning process and the learning outcome. Dubbed the 
Process/Outcome Evaluation Model (POEM) , the model expands and integrates 
existing evaluation models by providing the tools that can help decode, 
interpret, and assess not only what is learned, but also how the learning 
occurred. The evaluation matrix of POEM contains four categories of 
measurements: hard-outcome, hard-process, soft-outcome, and soft-process. By 
analyzing data from each measurement component singly, then collectively, 
evaluation using POEM becomes more comprehensive than traditional 
outcome-based testing strategies. (Contains 47 references.) (AEF) 
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The adoption of alternative pedagogical philosophies in the classroom has led to an increased use of 
technology to expand and enhance authentic, contextual learning environments. Correspondingly these new 
approaches have also led to a growing dissatisfaction with existing evaluation methodologies to evaluate knowledge. 

In this paper the Process/Outcome Evaluation Model (POEM) is proposed to guide in the development of more 
holistic evaluations of both the learning process and the resultant outcomes of that process. POEM consists of four 
components that employ multiple evaluation techniques and strategies resulting in a composite assessment of the 
totality of a learning experience. 



Introduction 

Traditionally, evaluation consists of a measurable test of learner knowledge, usually based on specified or 
implied learning objectives (Carey, 1988). On test day the learner is placed into an artificial testing environment, 
with no access to resources, and asked to answer a series of questions. Test results are calculated, compared to 
acceptable levels of performance, and assigned a grade or ranking. This measure of knowledge essentially evaluates 
only one dimension of knowledge; the ability of the learner to reproduce, recall, or recognize the teacher’s or test 
designer’s knowledge (Choi & Hannafin, 1995). In this scenario, what is learned is determined by what is measured 
(Jonassen, 1996) and since the assessment procedures are restrictive in scope, the learning outcomes and processes are 
commensurately limited. 

In order to reflect the multifaceted nature of learning, many educators believe assessment should encompass 
more than a single dimension of learning (Engel, 1994). This expanded view of evaluation is often referred to as 
authentic, performance-based evaluation (Darling-Hammond, 1994b). Traditional evaluation is described as secretive 
(Wiggins, 1989), expedient (Engel, 1994), and decontextualized (Jonassen, 1991), concentrating on discrete facts and 
rewarding convergent thinking. In contrast, authentic evaluation is seen as public (Wiggins, 1989), cumulative 
(Engel, 1994), and contextualized (Jonassen, 1991), integrating disciplines and enabling the learner to create original 
and unique problem solutions. However, even with authentic evaluation, educators are still confronted with the 
question of how to assess the totality of learning. 

This paper proposes an evaluation model based on the premise that evaluation strategies should reflect the 
full range of the experiences of learning. The model incorporates evaluation strategies that provide a composite 
picture of learning by examining both the learning process and the learning outcome. Termed the Process/Outcome 
Evaluation Model (POEM), the model expands and integrates existing evaluation models by providing tools that can 
help decode, interpret, and assess not only what is learned, but also how the learning occurred. 



Evaluation, Alternative Pedagogies, and Technology 

The continuous and cyclical nature of educational testing makes evaluation pervasive and significant for 
learners and educators. Even though tests serve multiple purposes, including assessing student knowledge, 
teacher/lesson effectiveness, and curriculum content (Carey, 1988), evaluation results most directly effect learners. 

Test scores are used to allow or deny access to educational opportunity. 

Learners are effected by decisions made based on test results that shape their educational future and by a 
curriculum that is dictated by test content. Since the focus is often achievement of specified test results, evaluation 
results can become the driving force behind educational strategies (Worthen & Spandel, 1991). Levels of test 
achievement frequently define what is taught (Haney & Madaus, 1989). 

While supposed or real deficiencies in basic skills or knowledge may be overcome through emphasis on 
traditional subjects such as reading, writing, and computation, critics of the current educational system argue that 
students are unable to apply knowledge and skills learned in school to nonschool situations (see, for example, 
Brandt, 1993). In other words, some educators question whether schools are teaching the skills necessary for the work 

“PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 




161 2 

BEST COPY AVAILABLE 



M. Simonsen 

U ~TOTHE EDUCATIONAL RESOURCES j 
INFORMATION CENTER (ERIC).” / 



of scientists, mathematicians, artists, engineers, writers, educators, and the other professionals (Resnick, 1987). 

Constructivist theories of learning recognize the learner as chief architect of knowledge creation through the 
application of unique experiences and beliefs to the learning process. Alternative pedagogies based on these 
assumptions focus on the individual while fostering skills in higher-order thinking and problem-solving strategies. 

Salient features of constructivist learning approaches include an emphasis on a) authenticity (Brown, 
Collins, & Duguid, 1989; Cronin, 1993); b) group work (Savery & Duffy, 1995; Slavin, 1991; English & Hill, 
1994); c) learner definition and control of the learning experience (Jonassen, Campbell, & Davidson, 1994; Kinzie & 
Sullivan, 1989); d) the teacher as guide, facilitator, coach (Cognition &Technology Group at Vanderbilt, 1993); e) 
divergent learning outcomes (Bransford, Sherwood, Hasselbring, Kinzer, & Williams, 1990); and f) the use' of 
supportive learning environments that are, in themselves, authentic, ecologically valid, and learner-centered (Krovetz, 
Casterson, McKowen, & Willis, 1993; Newmann & Wehlage, 1993; Stepien & Gallagher, 1993; Morrison & 
Collins, 1995; Wilson, 1995; Laszlo & Castro, 1995). Frequently, the use of ill-structured, unresolved social 
problems such as devising a solution to the nuclear waste problem form the basis of constructivist learning 
environments (Stepien & Gallagher, 1993) where learner solutions are evaluated not by their adherence or 
nonadherence to factual data, but by how well the solution solves the problem being investigated. 

Technology figures prominently in these learning environments since technology enables both knowledge 
exploration (Jonassen, 1988; Locatis, Letoumeau, & Banvard, 1989; Marchionini, 1988) and knowledge creation 
(Harris & Cady, 1988; Jonassen, 1988; Jonassen, 1996; Lehrer, Erickson, & Connell, 1994) without the traditional 
reliance on teacher and textbook as primary providers of knowledge. Technology is adaptable to less traditional 
educational purposes including individual learning, hypermedia development, and information exploration. In 
addition, technology can facilitate evaluation by gathering continuous information of learner progress, providing a 
readily usable storage device for evaluation results, and creating realistic microenvironments where learners are asked 
to solve problems that mirror reality. 

The incorporation of alternative pedagogical models has begun to shift the emphasis away from simple, 
one-shot evaluation measurements to a broader array of assessment methodologies. Alternative forms of evaluation 
tend to encourage the use of multiple vehicles for showcasing student achievement including portfolios, skill 
demonstrations, student performances, artistic works, and computer-related projects. 

Although much attention has been directed to alternative forms of evaluating student progress (Darling- 
Hammond, 1994a; Darling-Hammond, 1994b; Engle, 1994), educational evaluation, in practice, still generally 
focuses on only one dimension of learning; the learning outcome. Evaluating the learning process must also be an 
integral part of evaluation strategies (Choi & Hannafin, 1995; English & Hill, 1994; Jonassen, 1991; Jonassen, 
1996; Kumar, 1994; Kumar, Helgeson, & White, 1994; Marchionini, 1988; Webb, 1995). 

Evaluation of Learning Process and Learning Outcome 

Jonassen (1988) defines learning as the reorganization of cognitive structure, or, the expansion of the 
learner’s semantic network. This definition implies that learning produce divergent, not convergent learning 
outcomes. Further, if divergent learning best occurs when learners are able to incorporate new information into their 
preexisting semantic networks, then allowing learners to determine how best to learn may be preferable to over- 
structuring that experience for the learner (Kinzie & Berdel, 1990). 

One way learners structure their learning experience is by exercising control over various lesson attributes 
including instructional sequence, pacing, and options. Learner control removes the learner from a prescribed, 
regimented, predetermined lesson sequence and allows freedom of movement in and between lesson components. This 
freedom often enables the learner to structure the learning experience in a way that may best facilitate fusing new 
information with preexisting knowledge. 

Learning can be differentiated into learning processes, or how the learning occurs, and learning outcome, or 
what is learned. The strategies and mental processes used for learning differ from what is learned. Learning processes 
are internal, unique to the learner, and difficult to evaluate with traditional testing instruments (Kumar, et al„ 1994). 
Learning outcomes, on the other hand, can be externalized by the learner and are normally seen as a product of the 
learning situation. Learning outcomes can be as simple as the learner’s performance on a forced-choice test, or as 
elaborate as a learner-generated hypermedia program. Learning outcomes are generally more easily evaluated than 
learning processes since outcomes are tangible and processes tend to be subtle and often idiosyncratic. 

Separating “hard,” statistically measurable, reliable, and valid evaluation criteria from “soft,” interpretive, 
aesthetic, and affective evaluation criteria provides a classification system to evaluate outcome and process in two 
dimensions. The resulting evaluation matrix contains four categories of measurements: hard-outcome, haid-process, 
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soft-outcome, and soft-process. These four measurements comprise the framework of the Process/Outcome 
Evaluation Model (POEM) (see Figure 1). By analyzing data from each measurement component singly, then 
collectively, evaluation using POEM becomes more comprehensive than traditional outcome-based testing strategies. 
Evaluation takes on a broader perspective, investigating and synthesizing the totality of the learning experience. In 
the sections that follow, each quadrant of POEM will be described and examples of each evaluation type provided. 

Figure 1. The Process/Outcome Evaluation Model 
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Hard-Outcome Evaluation 

Learning outcomes are tangible representations of what the learner has learned. In today’s educational 
climate, a learning outcome can be a multiple-choice test, an essay, a collaborative oral report, a written report, an 
art project, or any number of products or performances that adequately measure mastery of the goals and objectives of 
the learning experience. 

The hard-outcome evaluation component will contain answers to questions that are driven by learning 
objectives and whether or not the learner achieved a satisfactory level of knowledge acquisition or demonstratable 
skill. A wide variety of tests can be employed ranging from forced-choice tests, to criterion checklists containing 
relevant skills for specific activities, to criteria for observational tasks, to problem-specific rubrics. 

Criterion- and norm-referenced tests are typical hard-outcome instruments. However, the tendency to envelop 
testing situations in secrecy where test takers receive little information prior to the test and minimal feedback on test 
results hinders open discussion about test content and test outcomes (Wiggins, 1989), thereby limiting their value as 
evaluative measurements (Berliner, 1992). 
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An added dimension to hard-outcome evaluation is the use of multiple evaluators, resulting in juried 
evaluations of products, such as portfolios. Along with improved reliability and validity produced by multiple raters, 
learners can be exposed to multiple sources of feedback and criticism. In this way, evaluation becomes a partnership 
where learners continue with the learning process as cognitive apprentices. 

Hard-Process Evaluation 

Hard-process evaluation consists of gathering and analyzing objective, quantifiable data on how learning 
occurs. Examples include counting the number of times a learner attempts a problem solution; tallying the number 
of interactions between students; tracking how often students follow prompts, guidance, and suggestions; and 
recording the learner’s path through instructional materials. 

Teachers often gather hard process data formally or informally by tabulating learning conditions and learning 
readiness in their classrooms. A teacher may categorize the types of questions being asked in order to induce where 
learners are in the learning process. Constant reference to printed material, quizzical looks, and student conversations 
are other examples of actions educators can count to aid in evaluating the learning process. The use of technology 
can greatly facilitate hard-process evaluation. If, for example, hypermedia is selected as part of the instructional 
medium, audit trail data provides insight into questions concerning the breadth and depth of a learner ‘s path, time 
spent at each learning activity, number of times an activity is accessed, and how often program features such as 
notebooks, help screens, and scrapbooks are accessed. Collection of hard process data not only contributes to 
individual learner profiles, but also assists in analyzing lesson strengths and deficiencies. 

The collection of quantifiable data for path analysis is relatively simple within a hypermedia environment. 
However, path analysis tends to produce large amounts of data per individual learner, and unmanageable datasets for 
groups of learners. If audit trail data is used to analyze the learning process, then methods to organize the massive 
amounts of audit trail data collected must be defined. 

Misanchuk and Schwier (1992) offer four ways to represent quantitative audit trail data including audit trail 
trees, where multiple paths are successively layered on a side-ways tree-like structure, visually reflecting 
commonalities between learner paths, with the most commonly accessed paths appearing as the thickest branches. 

Flynn (1994) used audit trail data to discern “navigational styles” of learners. By recreating and comparing 
paths across learners, four broad categories of navigational styles were identified: a) browsers, b) explorers, c) 
investigators, and d) novices. Each navigational style was defined by how learners accessed program components. For 
example, a learner who consistently moved randomly in the hypermedia lesson was classified as a “novice,” whereas 
learners who went back to screens or activities already accessed and then systematically delved deeper into a specific 
category were classified as “investigators” (Flynn, 1994). 

Based on actual user interaction, audit trail data is valid, measurable, and reliable. The hard-process 
evaluation strategies based on hypermedia-related audit trail path analysis provide meaningful measurements about 
the learner’s journey through hypermedia in a quantifiable way. 

Technology-based observations accomplished through adaptive instructional systems rely on the computer’s 
pre-programmed ability to interpret learner responses to allow for diagnostic assessment and continuous advisement 
(Tennyson, 1984). This approach results in lesson sequences tailored to the individual learner. Lesson options are 
determined by a historical interpretation of what a specific learner has done and is likely to do. Adherence to and 
deviation from prescribed sequences can be tracked and analyzed as part of hard- process evaluation strategies. 

Soft-Outcome Evaluation 

Soft-outcome evaluation encompasses the subjective assessment of learner-generated products and the 
experience in total. These evaluation strategies include criterion checklists, expert appraisal, and self-reflection. In 
general, for soft-outcome evaluation to be valid, products must reflect the outcome of solving authentic tasks. A 
portfolio containing an array of products focusing on the solution to a problem is one example of the type of 
outcome evaluated through soft-outcome strategies. 

Soft-outcome evaluation relies on the expertise and complexity of human evaluative interactions. 
Engagement in ongoing, evaluation-driven dialogue needs to occur between learner and evaluator, between learner and 
groups of raters and critics, between learner and peers, and between learner and experts. These interactions should 
assess learner conceptual understanding through the provision of meaningful and pertinent guidance and advice. 
Learners and their peers, for example, should establish ways to build on each other’s strengths while striving to 
improve areas of weakness. Learners also need to be given time and encouraged to reflect on the learning activity, 
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focusing on what worked and what did not, unresolved questions, and how this learning experience may transfer to 
other problems. 

Historically, the competitive nature of schooling stifles evaluative dialogue. Learners are usually motivated 
to out-perform their peers, not to help them do better. The use of strategies such as cooperative and collaborative 
learning can help foster discussion in the classroom about the importance of understanding and defining the problem 
to the solved, attributes of problem solving strategies, consequences of planning or not planning activities, and the 
efficiency and effectiveness of various problem solutions. These discussions, while evaluative in nature, do not 
emphasize the correctness of a problem solution, but rather stress the strengths and weaknesses of solutions when 
viewed from multiple perspectives. 

Another emphasis of soft-outcome evaluation is assessing learner ability to move onward. Questions to be 
pursued in this regard need to be contextually sensitive, reflective of a particular learning activity. Has the learner 
demonstrated the desire or inclination to broaden the inquiry? Can the learner actively and effectively debate a 
position or belief? What motivates this particular learner? What has the learner done to self-critique his/her 
experience? Combining observation and conversation to derive answers to these questions gives educators a solid 
foundation from which to draw inferences about learning outcomes. 

Soft-Process Evaluation 

Soft-process evaluation strategies employ qualitative measurement techniques. The process of learning is 
internal and often difficult to evaluate without involving the learner in some type of dialogue about what he or she is 
doing, or has done while in a learning situation and why these activities were chosen. For example, correlating 
learner activity with stated intentions during a problem-solving activity can help identify rationales for learning 
decisions and clarify the learning process (Kumar, et al., 1994). 

Kumar (1994) focused on how hypermedia can be used as a “tool for following the learner's cognitive 
processes” (p. 60), and thus evaluate how learning occurs. Kumar (1994) suggests using the navigational path as the 
basis for understanding the learner’s problem-solving process since it reflects all the choices and decisions made by 
the learner while solving a problem. 

Soft-process evaluation strategies help assess what learners are doing while actively engaged in a learning 
situation. Observational skills are critical in order to better discern and interpret activities such as student 
interactions; the use of ancillary materials, such as reference books and notepads; and the iterations involved in 
building and refining skills, such as when students create multiple versions or drafts of a project or paper en route to 
a finished product. Observing learner behaviors, talking with learners about what is observed, and encouraging 
learners to identify when, where, and why they made specific choices all contribute to evaluation. In this sense, 
learner behavior is used to infer how the learner is learning. 

Additionally, soft-process evaluation strategies are congruent with knowledge engineering, where post- 
instructional changes in a leamer‘s semantic network are discussed and interpreted (Jonassen, 1996). Comparing a 
learner’s semantic network over time highlights the complexity and interrelatedness of ideas and concepts. 
Computerized tools are available that create a graphical representation of a semantic network, often called a 
“cognitive map” (Fisher, 1990; Jonassen, 1996). A cognitive map represents ideas or concepts as nodes filled with 
text and/or graphics, and relationships or “links” between ideas and concepts are represented by appropriately labeled 
lines (Jonassen, 1996). Cognitive maps are unique to each learner and, since they represent a specific learner’s 
knowledge, they must be created by the learner whose knowledge they represent. This knowledge engineering 
technique may be most beneficial if the learner is able to dynamically alter his/her cognitive map while in a learning 
situation. 

Harmon (1992) theorizes that navigational strategy exhibited by a learner is related to his or her semantic 
network and that the navigational strategy employed by a learner is influenced by preexisting knowledge. Harmon 
and Dinsmore (1994) examined how learners “form semantic association^]” in a hypermedia environment. They 
found that the types of associations, or links, made by learners in hypermedia closely resembled the types of links 
that comprise semantic networking schemes (Harmon & Dinsmore, 1994; Fisher, 1990; Lambiotte, Dansereau, 
Cross, & Reynolds, 1989). 

The interrelationships between navigation, hypermedia, and semantic and cognitive representations may be 
best understood in terms of metacognition. Duell (1986) defines metacognition as knowledge about and regulation of 
the process of knowing. As learners struggle with choosing appropriate problem-solving strategies, planning 
activities, and monitoring progress toward possible solutions, soft-process evaluation can be instrumental in making 
these metacognitive skills more apparent and available to the learner. Suggestions can be given, guidance offered, and 
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questions asked that will focus the learner’s attention on what he/she is doing. Soft-process evaluation enhances 
metacognitive awareness by helping learners direct attention to how learning occurs with the ultimate aim of helping 
learners learn about themselves as learners. 

Conclusion 

Viewing POEM as a continuum, reliability and predictive validity increase as evaluation strategies move 
from “soft” to “hard” categories. POEM stresses an equilibrium between objective/quantifiable and 
subjective/qualitative evaluation approaches. The value of the model is its depiction of a holistic framework for 
evaluation. 

Evaluation must move away from its competitive nature and toward a more humane treatment for 
measuring student knowledge and ability. Unfortunately, teachers-in-training get little or no formal instruction on 
comprehensive evaluation techniques and strategies. 

Following the direction of Eisner’s (1983) educational connoisseurship and educational criticism model, 
teachers and learners alike can benefit from embracing a critical connoisseurial outlook and understanding of 
evaluation and how best to measure learning experiences in meaningful ways. Becoming comfortable with these 
techniques takes time and practice, coupled with mutual respect between all involved in the evaluation process. 
Honest and open discussions with multiple evaluators where the learner is encouraged to question opinions and 
critically judge his/her own work will build better learners, better evaluators, and more humane and authentic 
learning environments. 

Evaluation profoundly impacts American education. Throughout the schooling experience, learners are 
subjected to countless tests, quizzes, and examinations that purport to measure what has been learned. According to 
many educators the emphasis on tests has usurped the mission of schools to such an extent that the tests themselves 
control what teachers teach. Of course the problem is not this tendency to “teach to the test,” but the quality and 
unidimensionality of the tests themselves (Wiggins, 1989). 

POEM provides a framework to encourage educators and instructional designers to incorporate a variety of 
evaluation components within constructivist learning strategies. POEM reflects the importance of multiple 
evaluation strategies and proposes ways technology can be used to supplement and complement traditional evaluation 
efforts. 
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